HIVE-29177: Implement default Catalog selection #6088

zhangbutao · 2025-09-19T10:02:51Z

What changes were proposed in this pull request?

This PR first implements the functionality for clients to switch catalogs, such as SET CATALOG testcat;. Subsequently, created databases or retrieved database lists will all reside under the testcat catalog.
The primary changes in this PR involve modifying database-related DDL interfaces to ensure that operations like CREATE/ALTER/DROP DATABASE respect the client's current catalog.
You can check the q file catalog_database.q to see some syntax about cat.db. Such as:

-- CREATE DATABASE in new catalog testcat by catalog.db pattern
CREATE DATABASE testcat.testdb_1;
-- Switch database by catalog.db pattern
USE testcat.testdb_1;

-- Drop database by catalog.db pattern
DROP DATABASE testcat.testdb_1;

This PR added some TODO catalog labels, as some catalog tasks need to be fixed in other tickets. You can search TODO catalog to find which task should be done later.

Additionally, this PR only addresses the interfaces for switching catalogs and databases, and does not cover interfaces related to cat.db.table. Therefore, even if the catalog and database are switched, the current client will still query tables from the default hive catalog. The code related to cat.db.tbl (part of the work in HIVE-29177) will also involve many table-related interfaces, which presents certain refactoring challenges.

Why are the changes needed?

We need to make database work with catalog to support multiple catalogs in client simultaneously.

Does this PR introduce any user-facing change?

No. Users still can use classical syntax such as use dbname.

How was this patch tested?

mvn test -Dtest=TestMiniLlapLocalCliDriver -Dqfile='catalog_database.q' -Dtest.output.overwrite=true -pl itests/qtest -Pitests

sonarqubecloud · 2025-10-21T09:26:40Z

Quality Gate passed

Issues
106 New issues
0 Accepted issues

Measures
0 Security Hotspots
0.0% Coverage on New Code
0.0% Duplication on New Code

See analysis details on SonarQube Cloud

deniskuzZ · 2025-10-23T09:21:37Z

common/src/java/org/apache/hadoop/hive/ql/ErrorMsg.java

  RESOURCE_PLAN_NOT_EXISTS(10418, "Resource plan {0} does not exist", true),
  INCOMPATIBLE_STRUCT(10419, "Incompatible structs.", true),
-  OBJECTNAME_CONTAINS_DOT(10420, "Table or database name may not contain dot(.) character", true),
+  OBJECTNAME_CONTAINS_DOT(10420, "Catalog or table or database name may not contain dot(.) character", true),


maybe use comma: Catalog, database or table names

deniskuzZ · 2025-10-23T09:33:13Z

ql/src/java/org/apache/hadoop/hive/ql/ddl/catalog/use/SwitchCatalogDesc.java

+
+    private final String catalogName;
+
+    public SwitchCatalogDesc(String databaseName) {


catalogName ?

deniskuzZ · 2025-10-23T09:47:56Z

ql/src/java/org/apache/hadoop/hive/ql/ddl/database/create/CreateDatabaseDesc.java

+  public CreateDatabaseDesc(String catalogName, String databaseName, String comment, String locationUri, String managedLocationUri,
      boolean ifNotExists, Map<String, String> dbProperties) {
-    this(databaseName, comment, locationUri, managedLocationUri, ifNotExists, dbProperties, "NATIVE", null, null);
+    this(catalogName, databaseName, comment, locationUri, managedLocationUri, ifNotExists, dbProperties, "NATIVE", null, null);


for the external catalogs dbtype should be "REMOTE", right? It seems that we always hardcode "NATIVE" and below condition never returns true

if (dbtype != null && dbtype.equalsIgnoreCase("REMOTE")) {

for the external catalogs dbtype should be "REMOTE", right?

I don't think so. REMOTE dbtyp was introduced by HIVE-24396 data connector. If we implement all tasks about federated catalog, we can deprecate data connector. Because the data connector can only map external databases to internal databases by manually creating remote databases one by one to query external data sources, like this:
CREATE REMOTE DATABASE db_map_mysql USING mysqlconnector with DBPROPERTIES("connector.remoteDbName"="testmysqldb");

whereas the federated catalog can retrieve all external databases at once without the need to manually create remote databases, once we create a catalog.

It seems that we always hardcode "NATIVE" and below condition never returns true

If you create a remote database with a specified dataconnector, this condition will return true.

should we introduce a flag like native/non-native similar to tables?

deniskuzZ · 2025-10-23T09:54:13Z

ql/src/java/org/apache/hadoop/hive/ql/ddl/database/drop/DropDatabaseAnalyzer.java

+      throw new SemanticException(ErrorMsg.CATALOG_NOT_EXISTS, catalogName);
+    }
+    String databaseName = catDbNamePair.getRight();
+    Database database = getDatabase(catDbNamePair.getLeft(), catDbNamePair.getRight(), !ifExists);


why not use local vars?

getDatabase(catalogName, databaseName, !ifExists)

deniskuzZ · 2025-10-23T10:03:14Z

ql/src/java/org/apache/hadoop/hive/ql/ddl/database/unlock/UnlockDatabaseAnalyzer.java

    // Unlock database operation is to release the lock explicitly, the operation itself don't need to be locked.
    // Set the WriteEntity as WriteType: DDL_NO_LOCK here, otherwise it will conflict with Hive's transaction.
-    outputs.add(new WriteEntity(getDatabase(databaseName), WriteType.DDL_NO_LOCK));
+    outputs.add(new WriteEntity(getDatabase(catalogName, databaseName, true), WriteType.DDL_NO_LOCK));


HIVE_LOCKS table doesn't have catalog column.
Maybe some operation like Unlock/Lock DB, ShowDbLocks should only be supported by NATIVE DBs?

HIVE_LOCKS table doesn't have catalog column.

Currently, this only retrieves a specific database under a particular catalog and does not involve any lock/unlock operations on the catalog.
However, I believe future development should consider implementing lock/unlock operations for catalogs, though I haven't yet identified the specific use cases. It might be necessary during cross-catalog federated queries.

Maybe some operation like Unlock/Lock DB, ShowDbLocks should only be supported by NATIVE DBs?

To be honest, I haven't fully thought this through. Locking/unlocking databases in external catalogs (such as a MySQL JDBC catalog) doesn't seem very meaningful, as these lock operations won't take effect on the MySQL source side. However, I believe allowing locks on databases in external catalogs in Hive might have some significance, particularly in scenarios involving cross-catalog and cross-database operations, to ensure task transactional consistency. Therefore, let's further explore this issue when we work on cross-catalog queries. An already created ticket, HIVE-29242, can be used to track this matter.

deniskuzZ · 2025-10-23T10:07:00Z

ql/src/java/org/apache/hadoop/hive/ql/ddl/table/lock/show/ShowLocksOperation.java

      throw new HiveException("New lock format only supported with db lock manager.");
    }

+    // TODO catalog. Need to add catalog into ShowLocksRequest. But ShowLocksRequest doesn't have catalog field.


see https://github.com/apache/hive/pull/6088/files#r2454590955

Replied in #6088 (comment)

deniskuzZ · 2025-10-23T10:09:41Z

ql/src/java/org/apache/hadoop/hive/ql/exec/repl/ReplLoadTask.java

    Map<String, String> props = new HashMap<>();
    props.put(READONLY, Boolean.TRUE.toString());

+    // TODO catalog. Need to double check the actual catalog here. Depend on HIVE-29278


don't think we can replicate not NATIVE DBs

You are right. Some task can only do in NATIVE DBs. Like this ReploadTask.

This TODO is added to verify if the correct native catalog name is obtained. BTW, besides the default hive native catalog, we still need to support other multiple native catalogs.

As for external catalogs, we need to skip those tasks that are exclusive to native catalogs. However, the specific types of external catalogs (e.g., JDBC, REST) have not yet been defined, and this part of the work can be refined and completed subsequently. HIVE-29243 can be used to track this matter.

deniskuzZ · 2025-10-23T10:13:18Z

ql/src/java/org/apache/hadoop/hive/ql/exec/repl/incremental/IncrementalLoadTasksBuilder.java

-    AlterDatabaseSetPropertiesDesc alterDbDesc = new AlterDatabaseSetPropertiesDesc(dbName, mapProp,
-        new ReplicationSpec(replState, replState));
+    // TODO catalog. Need to double check the actual catalog here. Depend on HIVE-29278
+    AlterDatabaseSetPropertiesDesc alterDbDesc = new AlterDatabaseSetPropertiesDesc(HiveUtils.getCurrentCatalogOrDefault(conf),


@harshal-16 could you please validate if this is ok?

Just checked the details in the main JIRA: https://issues.apache.org/jira/browse/HIVE-28879
As ACID tables can only reside in Hive (default), other catalogs don't make sense in case of replication here
Also, during dump operation itself, it filters only default catalog here:

hive/ql/src/java/org/apache/hadoop/hive/ql/exec/repl/ReplDumpTask.java

Line 923 in d2bd353

new CatalogFilter(MetaStoreUtils.getDefaultCatalog(conf)),

thanks for checking, @harshal-16 ! what if we register multiple external HMS catalogs?

Oh, then I think we will need to update the incremental dump filter also to make it consistent with the load.

HIVE-29278 is used to track this matter. I think we can make the rep work with multiple native hms catalog.

deniskuzZ · 2025-10-23T10:17:47Z

ql/src/java/org/apache/hadoop/hive/ql/lockmgr/HiveTxnManagerImpl.java


-    HiveLock lck = lockMgr.lock(new HiveLockObject(dbObj.getName(), lockData), mode, true);
+    // Using the catalogName@databaseName format to uniquely identify a database.
+    HiveLock lck = lockMgr.lock(new HiveLockObject(catName + "@" +dbObj.getName(), lockData), mode, true);


oh, nice hack! however, maybe we should use "." ?
I think it might actually work. i am not sure if show locks would handle this change, might need some tweaks:

show locks db1 show locks cat1.db1

note, might be problematic if CU has multiple HMS versions on a cluster

nit: please add space + dbObj.getName()

deniskuzZ · 2025-10-23T10:24:20Z

ql/src/java/org/apache/hadoop/hive/ql/lockmgr/HiveTxnManagerImpl.java

      throw new HiveException("Database " + dbName + " does not exist ");
    }
-    HiveLockObject obj = new HiveLockObject(dbObj.getName(), null);
+    HiveLockObject obj = new HiveLockObject(catName + "@" +dbObj.getName(), null);


deniskuzZ · 2025-10-23T10:28:12Z